The Binomial Distribution

Table of Contents

• 1  The Binomial Distribution
  • • 1.0.1  Recall many coin flips problem
    • 1.0.2  What is p(X=k)?
    • 1.0.3  A Corollary - Bernoulli trial and Binomial Distribution
    • 1.0.4  Maximum Probability
    • 1.0.5  Mean of binomial distribution
    • 1.0.6  Variance of binomial distribution
    • 1.0.7  Conclusion

Recall many coin flips problem

We already saw, as the number of flips grew larger, the distribution of X started to form a bell shaped pattern. Let us recollect that for a moment.

4 Flips - Tree

from graphviz import Digraph
from coinflipviz import draw_graph, get_combinations, get_combinations_consolidated, plot_combinations_consolidated

n_flips = 4

g = Digraph()
g = draw_graph(g, n_flips)
g

All possible sequences

combi_df = get_combinations(n_flips)
combi_df

	sequence	x
0	HHHH	4
1	THHH	3
2	HTHH	3
3	TTHH	2
4	HHTH	3
5	THTH	2
6	HTTH	2
7	TTTH	1
8	HHHT	3
9	THHT	2
10	HTHT	2
11	TTHT	1
12	HHTT	2
13	THTT	1
14	HTTT	1
15	TTTT	0

Consolidated no of sequences under X

final_df = get_combinations_consolidated(n_flips)
final_df

	x	n(x)	p(x)
0	0	1	0.0625
1	1	4	0.2500
2	2	6	0.3750
3	3	4	0.2500
4	4	1	0.0625

Graphs

plot_combinations_consolidated(final_df)

What is p(X=k)?

Let k =2, that is, X = 2, the no of heads being 2 in a sequence. From above table/ LHs graph we could see, there are 6 sequences with k = 2 heads. We already know, total no of sequence could be given by $2^4 = 16$.

So,
$ p(X=k) \ = p(X=2) \ = \ \dfrac {\text {No of seq with 2 heads}}{\text {Total no of seq}} \ = \ \dfrac {6}{2^4} = 0.375 \tag{1} $

Via formula

In fact we have a formula hidden in our observation. Note that, the no of heads in sequence is a typical combinatorics problem. The solution can be obtained by Combination formula. For example, out of 4 possible placements of a sequence(resulting after all flips), what are the number of occurances where, 2 placements are occupied with heads?

$ \text {No of seq with 2 heads out of 4 placements} \ = \ \dfrac {n!}{k!(n-k)!} \ = \ \dfrac {4!}{2!(4-2)!} \ = \ 6 \tag{2} $

Each of those sequences has a joint probability if individual probabilities are known. Assuming equal probabilities for both head and tail (that is 0.5 each), we could compute total joint probability as follows.

$ \text {Probability of 2 heads and 2 tail of any one sequence} \ = \ p \times p \times \ q \times q \ = \ (0.5) \times (0.5) \times (0.5) \times (0.5) \ = \ (0.5)^2(0.5)^2 \tag{3} $

Combining ${2}$ and ${3}$,
$ \text {Probability of all sequences with 2 heads out of 4 placements} \ = \ 6 \times \ (0.5)^2(0.5)^2 = 0.375 \tag{4} $

This could also become evident, if we re write equation ${1}$ as below.

$$ p(X=k) \ = p(X=2) \ = \ \dfrac {6}{2^4} \ = \ 6 \times \Big( \dfrac {1}{2} \Big)^2\Big( \dfrac {1}{2} \Big)^2 \ = \ 6 \times \ (0.5)^2(0.5)^2 = 0.375 $$

Generalizing,

we could write as for any k,

$$ \color {blue} {p(X = k) \ = \ \dfrac {n!}{k!(n-k)!}p^{k}q^{n-k} } \tag{5} $$

Try 10 flips

Let us check with 10 flips.. Obviously tree viz would be too large, so we will directly compute the graphs and compare.

n_flips = 10
final_df = get_combinations_consolidated(n_flips)
plot_combinations_consolidated(final_df)

Let us check for any k. Say, k = 5,

Graph:

From RHS graph, $ p(X=k) = p(X=5) = 0.24609375 $

Formula:

Plugging in n=10, k=5, p =0.5, q = 1-p = 0.5, in equation ${5}$
$$ p(X = k) \ = \ \dfrac {n!}{k!(n-k)!}p^{k}q^{n-k} \ = \ \dfrac {10!}{5!(10-5)!}(0.5)^{5}(0.5)^{10-5} \\ \\ = \ \dfrac {10 \times 9 \times 8 \times 7 \times 6}{5 \times 4 \times 3 \times 2 \times 1}(0.5)^{5}(0.5)^{5} \\ \\ = \ (252)(0.5)^{10} = 0.24609375 $$

A Corollary - Bernoulli trial and Binomial Distribution

The coin flip problem is a typical Bernoulli trial.

A Bernoulli trial is each repetition of an experiment with only 2 outcomes

We are often interested in the result of independent, repeated bernoulli trials, i.e. the number of successes in repeated trials.

1. independent - the result of one trial does not affect the result of another trial.
2. repeated - conditions are the same for each trial, i.e. p and q remain constant across trials. Hayes refers to this as a stationary process. If p and q can change from trial to trial, the process is nonstationary. The term identically distributed is also often used.

A binomial distribution gives us the probabilities associated with independent, repeated Bernoulli trials. In a binomial distribution the probabilities of interest are those of receiving a certain number of successes, r, in n independent trials each having only two possible outcomes and the same probability, p, of success. So, for example, using a binomial distribution, we can determine the probability of getting 4 heads in 10 coin tosses

Any random variable X with probability function given by

$$ \color {blue} {p(X=k; N,p) \ = \ \dfrac {N!}{k!(N-k)!}p^{k}q^{N-k} \ , \ X = 0,1,2,....N} $$

is said to have a binomial distribution with parameters N and p.

Courtesy: The Binomial Distribution

Maximum Probability

From 10 flips problem graphs above, one could easily observe, tha maximum no of sequences with k or maximum probability of that occurs when $k=5$ which is, $\dfrac {10}{2} = \dfrac {n}{2}$

Mathematically, the function $\dfrac {n!}{k!(n-k)!}p^{k}q^{n-k}$ becomes maximum when $k=\dfrac {n}{2}$

$ \dfrac {n!}{k!(n-k)!}p^{k}q^{n-k}\Bigg\rvert_{k=\dfrac{n}{2}} \ = \ \dfrac{n!}{2\Big(\dfrac{n}{2}\Big)!} $

Mean of binomial distribution

But where is this $\dfrac {n}{2}$ coming from? Well, that is the mean of our binomial distribution coin flip problem. The graphs simply maximized at the mean $E(X)$.

Emperical Proof

Let us try to reverse engineer a little bit. Let us consider again no of flips = 4.

n_flips = 4
final_df = get_combinations_consolidated(n_flips)
plot_combinations_consolidated(final_df)

In plain english, from above LHS graph, 0 heads occur 1 time, 1 heads occur 4 times, 2 heads 6 times and so on.

So What is the mean of no of heads in this experiment? In other words, mean value for X. We know from graph it is 2, but let us find it mathematically there by deriving general formula $E(X)$

From LHS graph, we know the frequency of X for each possible X value. n(X=0) = 1, n(X=1) = 4 etc.

So by general mean formula applicable to any regular data series,

$$ E(X) = \dfrac {0 \cdot 1 + 1 \cdot 4 + 2 \cdot 6 + 3 \cdot 4 + 4 \cdot 1}{1+4+6+4+1} \\ \\ = \dfrac {0 + 4 + 12 + 12 + 4}{16} = \dfrac {32}{16} = 2 $$

Let us rewrite from derived solution,..

$$ \begin{array}{l} E(X) = \dfrac {0 + 4 + 12 + 12 + 4}{16} \\ \\ = \dfrac { \sum\limits_{k=0}^4 X_k \cdot n(X_k) }{2^4} \\ \\ = \sum\limits_{k=0}^4 \dfrac {X_k \cdot n(X_k)}{2^4} \\ \\ \color {blue} {= \sum\limits_{k=0}^4 X_k \cdot p(X_k)} \\ \\ \end{array} $$

Note, $X_k$ is just $X=k$.

We could also expand this to again arrive at our solution, just to clarify how things unfold to and fro.

$ \require{cancel} \begin{array}{l} E(X) = \sum\limits_{k=0}^{4} X_k \cdot p(X_k) \\ \\ = 0 \cdot p(X=0) + 1 \cdot p(X=1) + 2 \cdot p(X=2) + 3 \cdot p(X=3) + 4 \cdot p(X=4) \\ \\ = 0 + 1 \cdot \binom{4}{1}p^{1}(1-p)^{3} + 2 \cdot \binom{4}{2}p^{2}(1-p)^{2} + 3 \cdot \binom{4}{3}p^{3}(1-p)^{1} + 4 \cdot \binom{4}{4}p^{4}(1-p)^0 \\ \\ = 0 + 4 \cdot p^{1}(1-p)^{3} + 12 \cdot p^{2}(1-p)^{2} + 12 \cdot p^{3}(1-p)^{1} + 4 \cdot p^{4} \\ \\ = 0 + 4p(1-p)\Big\{ (1-p)^2 + 3p(1-p) + 3p^2\Big\} + 4p^4 \\ \\ = 0 + 4p(1-p)\Big\{ (1-p)^2 + 3p - \cancel{3p^2} + \cancel{3p^2} \Big\} + 4p^4 \\ \\ = 0 + 4p(1-p)\Big\{ 1 - 2p + p^2 + 3p \Big\} + 4p^4 \\ \\ = 0 + 4p(1-p)\Big\{ 1 + p + p^2 \Big\} + 4p^4 \\ \\ = 0 + 4p \Big\{ 1 + \cancel{p^2} + \cancel{p} - \cancel{p} - p^3 - \cancel{p^2} \Big\} + 4p^4\\ \\ = 0 + 4p \Big\{ 1 - p^3 \Big\} + 4p^4 \\ \\ = 4p \end{array} $

The 4 in above equation is number of flips.

Proof by Binomial theorem

According to the theorem, it is possible to expand any power of x + y into a sum of the form

$$ (x+y)^{n}=\sum _{k=0}^{n}{n \choose k}x^{k}y^{n-k}. $$
For n = 3, this would be,

$$ \begin{array}{l} (x+y)^{3}=\sum _{k=0}^{3}{3 \choose k}x^{k}y^{3-k} \\ \\ = \binom {3}{0}x^{0}y^{3} + \binom {3}{1}x^{1}y^{2} + \binom {3}{2}x^{2}y^{1} + \binom {3}{3}x^{3}y^{0} \end{array} $$

$$ =y^3 + 3xy^2 + 3x^2y + x^3 \tag{6} $$

Note the coefficients: [1 , 3 , 3 , 1]. These are the binomial coefficients, and could be found in Pascal's triangle at the row corresponding to the value n

Suppose q denotes the probability of tails, q = 1 - p, then, we could rewrite equation ${6}$ as below

$ \begin{array}{l} E(X) = \sum\limits_{k=0}^{4} X_k \cdot p(X_k) \\ \\ = 0 \cdot p(X=0) + 1 \cdot p(X=1) + 2 \cdot p(X=2) + 3 \cdot p(X=3) + 4 \cdot p(X=4) \\ \\ = 0 + 1 \cdot \binom{4}{1}p^{1}q^{3} + 2 \cdot \binom{4}{2}p^{2}q^{2} + 3 \cdot \binom{4}{3}p^{3}q^{1} + 4 \cdot \binom{4}{4}p^{4}q^0 \\ \\ = 0 + 4pq^3 + 12p^{2}q^{2} + 12p^{3}q + 4p^4 \\ \\ =4p\Big\{ q^3 + 3pq^{2} + 3p^{2}q + p^3 \Big\} \\ \\ =4p\Big\{ (p+q)^3 \Big\} \ \ \text{using binomial theorem}\\ \\ \\ \end{array} $
Note, $p+q=1$ $$ \therefore \ \ E(X) = 4p $$

Generalization

Thus in general, we could calculate the mean of a binomial distribution of n flips as follows.

$$ \begin{array}{l} \color {blue} {E(X) = \sum\limits_{k=0}^nX_k \cdot p(X_k) = np} \tag{7} \end{array} $$

We had proven with specific number of flips, but this could be proved for any n mathematicallym with same approach.

Variance of binomial distribution

Emperical Proof

Let us revisit again graphs for no of flips = 4.

n_flips = 4
final_df = get_combinations_consolidated(n_flips)
plot_combinations_consolidated(final_df)

final_df

	x	n(x)	p(x)
0	0	1	0.0625
1	1	4	0.2500
2	2	6	0.3750
3	3	4	0.2500
4	4	1	0.0625

In plain english, from above LHS graph, 0 heads occur 1 time, 1 heads occur 4 times, 2 heads 6 times and so on.

So What is the variance of no of heads in this experiment? In other words, variance $\sigma^2$ for X. Let us find it mathematically by using general formula $Var(X)$

From LHS graph, we know the frequency of X for each possible X value. n(X=0) = 1, n(X=1) = 4 etc.

So by general variance formula applicable to any regular data series (given with frequency count of each X),

$$ Var(X) = \dfrac { (0-2)^2(1) + (1-2)^2(4) + (2-2)^2(6) + (3-2)^2(4) + (4-2)^2(1)}{2^4} \\ \\ = \dfrac {4+4+0+4+4}{16} = \dfrac {16}{16} = 1 $$

Let us rewrite from derived solution,..

$$ \begin{array}{l} Var(X) = \dfrac {0 + 4 + 12 + 12 + 4}{16} \\ \\ = \sum\limits_{k=0}^4 \dfrac {(X_{k}-\bar{X})^2n(X_k)}{2^4} = \sum\limits_{k=0}^4 (X_{k}-\bar{X})^2p(X_k) \end{array} $$

Thus, $$ \color {blue} {\sigma^2 = Var(X) = \sum\limits_{k=0}^n (X_{k}-\bar{X})^2p(X_k)} \tag {8} $$

Using equation $7$, $$ E(X) = \sum\limits_{k=0}^nX_k \cdot p(X_k) \\ \\ $$ $$ \therefore E(X^2) = \sum\limits_{k=0}^nX^2_k \cdot p(X_k) \tag {9} $$

Expanding equation $8$,

$$ \begin{array}{l} Var(X) = \sum\limits_{k=0}^n (X_{k}-\bar{X})^2p(X_k) \\ \\ = \sum\limits_{k=0}^n\Big\{ X^2_k-2X_k\bar{X} + \bar{X}^2 \Big\}p(X_k) \\ \\ \\ = \sum\limits_{k=0}^nX^2_kp(X_k) - 2\bar{X}\sum\limits_{k=0}^nX_kp(X_k) + \bar{X}^2\sum\limits_{k=0}^np(X_k) \\ \\ \end{array} $$

$$ \sum\limits_{k=0}^np(X_k) = 1 \because \text {that is total probability of our problem which is 1} $$

Using that and also equation $9$, $$ Var(X) = E(X^2)-2\bar{X}^2+\bar{X}^2 = E(X^2) - \bar{X}^2 $$

Thus, $$ \color {blue} {\sigma^2 = Var(X) = E(X^2) - [E(X)]^2} \tag {10} $$

This alternate form could be useful to deduce a simplified formula for variance of binomial distribution as follows.

For n=4, we could expand equation $9$ as below.

$$ \begin{array}{l} E(X^2) = \sum\limits_{k=0}^4X^2_kp(X_k) \\ \\ = 0 + 1 \cdot \binom{4}{1}p^1q^3 + 4 \cdot \binom{4}{2}p^2q^2 + 9 \cdot \binom{4}{3}p^3q^1 + 16 \cdot \binom{4}{4}p^4q^0 \\ \\ = 4pq^3 + 24p^2q^2 + 36p^3q + 16p^4 \\ \\ = 4pq \big\{\ q^2 + 6pq+9p^2 \ \} + 16p^4 \\ \\ = 4pq \big\{ q^2 + 2(3p)q + (3p)^2 \big\} + 16p^4 \\ \\ = 4pq(q+3p)^2 + 16p^4 \\ \\ = 4pq(1+2p)^2 + 16p^4 \\ \\ = 4pq(1+4p+4p^2) + 16p^4 \\ \\ = 4pq + 16p^2q + 16p^3q + 16p^4 \\ \\ \\ \text {Var}(X) = E(X^2)-[E(X)]^2 \\ \\ = \big \{ 4pq + 16p^2q + 16p^3q + 16p^4 \big \} - 16p^2 \\ \\ = 4pq + 16p^2(1-p) + 16p^3q + 16p^4 - 16p^2 \\ \\ = 4pq + \cancel{16p^2} - 16p^3 + 16p^3q+16p^4 - \cancel{16p^2} \\ \\ = 4pq + 16p^3q+16p^4-16p^3 \\ \\ = 4pq + 16p^3q + 16p^3(p-1) \\ \\ = 4pq + \cancel{16p^3q} - \cancel{16p^3q} \\ \\ = 4pq \end{array} $$

Generalizing, for binomail problem of n flips, p probability of X, and p+q=1,

$$ \color {blue} {\sigma^2 = Var(X) = \sum\limits_{k=0}^n (X_{k}-\bar{X})^2p(X_k) = E(X^2) - [E(X)]^2 = npq} \tag {11} $$

Try 20 flips

Let us check mean and variance now with 20 flips.. Obviously tree viz would be too large, so we will directly compute the graphs and compare.

n_flips = 20
final_df = get_combinations_consolidated(n_flips)
plot_combinations_consolidated(final_df, label=False)

We know, n = 20, p = 0.5, q = 0.5

Mean

$ \mu = E(X) = np = (20)(0.5) = 10 $

This is easily verifiable from above graph, that at X=10, we get maximum n(X) and also p(X).

Variance

$ \sigma^2 = Var(X) = npq = (20)(0.5)(0.5) = 5 $

Conclusion

We have derived the three main formula for a problem with binomial distribution (Coin toss). For n no of flips, with probability of heads p being 0.5 at each flip, and q = 1 - p, we could calculate

1. Probability of X = k

$$ \color {blue} {p(X=k; n,p) \ = \ \dfrac {n!}{k!(n-k)!}p^{k}q^{n-k} \ , \ X = 0,1,2,....n} $$

2. Mean

$$ \color {blue} {\mu = E(X) = np} $$

3. Variance

$$ \color {blue} {\sigma^2 = Var(X) = npq} $$